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ABSTRACT 

This study describes a method for providing easily 
understood information on the comparative instructional program 
effects of using multiple instructional products. The method was 
derived iri the context of a national evaluation of three 
instructional product systems each of which was used to teach 
readiness and initial reading skills to kindergarten students. 
Longitudinal data were gathered from 900 classes, with each class 
using one of the products. The results illustrate how valid 
information on instructional effects can be obtained and used for the 
purpose of comparative evaluation. Further, the results contradict 
the popular belief that the specific instructional resources used in 
an instructional program do not matter. Instructional outcomes 
retleat -t+^e specific skills and concepts emphasized in a given 
product system, and these differences in emphasis emerge when 
instructionally sensitive instrumentation is used. Of more 
importance, the study illustrates a methodology that can be used for 
what is referred to as empirical curriculiun inquiry. This is a way of 
identifying what students know, what school programs commonly and 
uniquely teach, and what students learn. Some implications of this 
methodology for school improvement efforts are discussed, 
(Author/BS) 
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PROGRAM-FAIR EOUCATIOMAL EVALUATION AND EMPIRICAL CURRICULUM INQUIRY 

Ralph A. Hanson and Jerry D. Bailey 

ABSTRACT 

Obtalninq an understandable description of what Is belnq 
accomplished in a school program Is an extremely difficult task using 
avanab e methods. Even more difficult Is obtaining comparative 
information on program effects when multiple '"^^^"^^^"/l ^ 
being used. This study describes a method for prov ding easily 
unde?stood Information on such comparative instruct ona effec s. The 
. method was derived In the context of a national evaluation of hree 
Instructional product systems each of which was used ^^^^^^^ 
and initial reading skills to kindergarten students. Longitudinal data 
were qathered from 900 classes, with each class using one of the 
produ^rs. ?he results Illustrate how valid information on instructional 
effects can be obtained and used for the purpose of comparative 
evaluation. Further, the results contradict the popular bel lef that the 
Tpeclfc instructional resources used in an Instruct onal Program does 
not matter. Instructional outcomes reflect the specific Is and 
conceots emphasized in a given product system, and these differences in 
emphas ^17,1 when instruct lonal ly sensitive -'"f J^^^^JJ^ °" 
Of'more Importance, the study Illustrates a methodology that ca be u ed 
for what Is referred to as empirical curriculum Ingulry. ^^is Is a way 
of Identlfylna what students know, what school, programs commonly and 
uniquely teach, and what students learn. Some Implications of this 
methodology for school Improvement efforts are discussed. 
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PROCWW-FAIR EOUCATIOMAL EVALUATIOM AMD EMPIRICAL CURRICULUM INQUIRY 
Ralph A. Hansbn and Jerry D. BaMey 

Accurate descriptive Information on the effects of Instructional 
products designed to fulfill the same or similar educational program 
requirements has often been sought but rarely obtained In practice. In 
spite of.the use of a variety of methodologies and Instrumentation, the 
bottom line always seems to be the same; no specific effects of 
comparable instructional products can be detected. These results have 
led some researchers to conclude that such differences simply do not 
exist, i.e., the instructional resources employed In educational programs 
play a minor role, if any, in terms of measureable schooling effects. 

As with many other broad conclusions about school I ng effects, this 
one has not met with universal acceptance. Llke'the proposition that 
schools have "no effect," It lacks credibility on a common sense basis. 
Consequently, when an opportunity to examine the proposition arose in the 
context of fulfilling other research requirements, we pursued it. 

Th. opportunity arose when the U.S. Office of Education Initiated a 
project under Title ill of the Elementary and Secondary Education Act. 
The project provided support for the purchase by schools of certain 
instructional product systems to be used In their kindergarten reading 
readiness program; during the school year. The acceptance of 

Title III funds obligated participating districts to provide USOE with 
program evaluation data. For those districts willing to participate In 
the study, SWRL agreed to provide Instrumentation and other logistical 
support to fulfill the USOE evaluation requirement. The majority of the 
districts receiving funds chose this option. 
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Consistent with these eondittonsi the Inquiry began with the limited 
objective of provldlnci an operational methodology to measure the Instruc- 
tional effects obtained with the use of the alternative products In the 
-kindergarten reading readiness Instruction, "fhe general approach was to 
obtf'itn assessment results and other Information from districts on the 
competing products of Interest. Such data can readily be used to 
generate the reports required to fulfill the Title III evaluation 
requirement for each district (see Hanson, Schutz, & Baileyi 1977). 

The data also can be used for the conduct of research on program 
fair evaluation issues If two related issues are addressed. The first Is 
whether measures of reading readiness proficiency obtafned at the end of 
the school year can be llnlced to measures of pupil status obtained prior 
to instruction and Implementation measures obtained during instruction 
for samples of classes using each of the products. Several prior SWRL 
inquiries Indicate this Is possible at least with certain products and 
under certain conditions (Hanson t Schutz, 1978; Hanson, Bailey, & 
Molina, 1980),^'H^^^-hs-fiosslble In this context, It was reasoned that 
such resultLwould allow ul to estimate both the Instructional 
sensitivity and magnitude of outcome proficiencies attained by users of 
each product. Put another way, these two kinds of information would 
indicate what Instructional outcomes were yielded using a product and how 
well pupils learned what was taught. 

Given that reasonable associations could be obtained between the 
operational use of a product and proficiency on Its own outcomes, the 
second "program fair" Issue could^be addressed. Stated as a question, it 
asks, "Can the Instructional sensitivity and magnitude of effects of an 
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Instructional product be detected on outcomes not referenced to that 
product?" This Is the central Issue In program-fair evaluation; to 
describe the Impact of several alternative products on the full range of 
outcomes that the products collectively address based on data from 
representative and comparable users. 

DESIGN AND PROCEDURES 
With agreement on several specific tasks and these general research 
qoals, the KRR Inquiry was launched. Districts In six states, using the 
four most popular kindergarten text series were designated as the Inquiry 
population. Each of these districts was receiving funds for the purchase 
of one of the following: 1) Alpha Time (A), published by New Dimensions; 
2) Beginning to Read, Write and Listen (B), published by Llpplncotti 3) 
Dlstar I Reading (D), published by SRA; M The Kindergarten Program (K), 

publ Ished by Ginn. 

Districts choosing to participate would receive all achievement 
tests and related Instrumentation; summary reports on achievement 
performance for pupils, classes, schools, and districts; and fuifiliment 
of the Title III grant evaluation requirement. SWRL would be responsible 
for instrument development; design and operation of an Information system 
to distribute, receive, process, and report Information to various 
audiences; and analyses of the data base for a series of RSD reports.^ 



r!nSiirfdrs!gJ,':il:?!-ls-V-edure^ 
the data base. That Information is summarized below to the extent 
necessary to make this document coherent and stand-alone. 
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Design and instrumentation Considerations 

The requirements of tiie evaluation sponsors, operational constraints 
on schools, limitations of time and resources, the general goals of the 
Inquiry and other factors all had an Influence on the formulation of the 
study design. While Identifying all such factors and tracing their 
1inl<ape to each of the design decisions Is probably Impossible, some of 
this conceptualization should be^noted. 

One Important consideration was that the Inquiry had to be carried 
out in ongoing school settings. This meant that the actual use of the 
instructional products would be fully under the control of the schools 
using them. While this was desirable In the sense that product use and 
effects could be expected to reflect typical rather than forced patterns, 
it did necessitate consideration of variability in product 
Implementation. 

To describe product Implementation in each class, an Instructional 
Information Sheet (IIS) was prepared referenced to each product. The 
sheet Included a common set of Items about Instructional time and 
emphasis given to various reading readiness topics and a set of questions 
about use of sped f lo-a|ft-oduct Instructional resources. In this study, 
only the Information from the latter on the extent of product 
Instructional resource use are Involved. 

Another Inportant design consideration is that all participants 

chose both the reading readiness product to use and whether to take part 
in the study. These factors negated any possibility of either equating 
product user groups In advance or even assuming that the full range of 

I 
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kindergarten pupil entry behavior were represented In each product user 
groups. To take such differences Into account two approaches were used. 
The first was t6 measure certain characteristics of pupils and schools 
prior to the start of the program. The Entry Survey, a common test of 
reading readiness consisting of 22 Individually administered items 
covering basic semantic skills (Coker S Legum, 1972; 197M was used for 
this purpose. Teachers in each class were asked to administer this test 
to pupils on an individual basis prior to the start of Instruction. 

The second was to measure a set of biosoclal variables derived from 
the School Information Sheet (SIS). It provides Information on 
characteristics such as Income level, ESEA Title I eligibility, ethnlt- 
raclal background, etc., on the pupils attending each school. 

Another design consideration was the primary unit of analysis to be 
used In planning and analyzing the Inquiry data base. Both the nature of 
the schooling process, particularly In the early elementary grades, and 
earlier SWRL research on Instructional product Implementation suggested 
the classroom as the primary unit of analysis (Wiley & Bock, 1968; Hanson 
& Schytz, 1978). Briefly, the rationale for this choice Is that In the 
actual delivery of Instruction In an elementary school program such as 
kindergarten reading readiness, the primary source of variance In 
Implementation Is at the class level. Decisions made at the class level 
regarding scheduling of Instructional activities, the materials and 
procedures to be used, configurations of Instructional groups, etc., tend 
to be by far the most direct and significant Implementation elements and 
have been found to relate directly to Instructional product proficiency. 



Further, while these classrooms' implementation behaviors reflect in part 
the policies and practices of the schools and districts In which they 
operate, they tend to be most directly affected by specific classroom 
factors. ^ 

Another design factor of importance Is the provisions for measuring 
pupil proficiency at the end of the school year. Prior conceptual work 
on measuring the Instructional effects of alternative products (I.e., 
Wolf, 1968; Popham, 1963; Hajer & Sullivan, 1970; Shoemaker, 1972) all 
suggested that some or all or the following steps to be taken In 
preparing program-fair tests. 

• Obtain behavioral objectives^ for each product to be evaluated. 

• Analyze the composite set of objectives from all products to 
Identify the common and unlgue objectives for each product. 

• Develop measures of each conpnon and unlgue objective. 

/ 

• Generate Items to measure each outC(»iie and administer them In 
classes using each of the products to be evaluated. 

This general strategy departed from prevailing comparative assessment 

practices In recognizing that the effects of one Instructional product 

may or may not be reflected on a single, common measure such as a 

standardized achievement test. However, It provided no guidance for 

resolving the Issue of how to determine common and unique objectives for 

a given set of educational products. Rather than ignoring this Issue or 

assuming that the common ^nd unlgue objectives could be Identified on an 

analytical basis alone, a way of empirically Identifying the common- and 

unique effects of a product was sought. 
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The measurement strategy adopted was to generate test Items 
referenced directly to th« product's Instructional resources. Opera- 
tional ly,. this task was accomplished through detailed analytical work on 
the Instructional materials and procedures. These analyses were carried 
out separately on each product and resulted In tables that list each 
concept or skill presented, the format used to present Instruction on It, 
and some Indication of the amount It would be practiced under typ.cal 
use. 

From these specifications, prototype Items referenced to each 
concept or skill resulted. TheU prototype Items organized under the 
published outcome descriptions which accompanied each product were then 
reviewed by the product's publisher for additions and deletions. After 
Incorporating the publisher's coirments, the full Item pool was completed. 
It was designed to Include Items reflecting each format used In 
Instruction and most, if not all, of the specific elements taught. 

The final lists of outcome ^reas are reproduced In Appendix A. For 
each product outcome a brief definition and the number of Items generated 
under the area Is Indicated and an Instructional area number Is given. 
The latter refers to one of a set of 12 broader Instructional areas under 
which the outcomes of all four products were further organized. Each 
Instructional area Is defined In Figure 1 and the specific outcomes of 
each of the four Instructional products which fall under It are 
Indicated. 

Note that for some Instructional areas all the products are 
represented by one or more outcomes, while under other Instructional 
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areas as few as one outcome and product appear. These reflect broad 
differences In Instructional emphasis between products. Examination of 
the specific entries In this figure provides preliminary comparative 
Information on these differences In emphasis without referring to the 
relative effectiveness of the products In affecting learning In the area, 



Insert Figure 1 about here 



I 

Assessment Development 

The Item pools described In Appendix A were used as the elements for 
building three long (maxi) and three short (mini) test forms referenced 
to each pro(luct. Each maxI (or long form) was designed t© test all 
outcomes of the product. Taken together, the three maxI forms exhausted 
the Item pool for the product. The m!nl forms were designed to measure 
only the major outcomes of a product. They typically Included about 
one-half as many Items as the maxl forms. Taken together, however, the 
three mini forms covered all major outcomes of a product. They did not, 
however, exhaust all Items In all the pools. 

The mint and maxl test forms were combined for administration. 
Thus, each class received a test composed of two parts; a maxl form 
referenced to the product being used In the class and a mini form 
referenced to one of the other comparison products. This assessment 
design Is a modified matrix sampling approach In which the mini forms 
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from comparison products are distributed across the users of each 
product. 2 

Participation 

The flow of study instrumentation went from SWRL through a 
coordinator, designated by each district, to classrooms and then back 
through the coordinator to. SWRL. This arrangement had several 
advantages. It allowed the pupil and teacher sources of information to 
remain anonymous outside the district. The coordinator assigned and 
maintained a class numbering system within the district. Within classes, 
the teacher maintained a pupil numbering system. In addition to 
monitoring the receipt and return of materials, the coordinator also 
provided support and assistance where needed In the completion of 
materials and Interpretation of results. . 

The data on participation, measured in terms of data components 
con^leted, Indicate that district coordination and support were excellent 
in this Inquiry. Over 1,000 classes, representing about m of the 
eligible districts in the six states. I.e., those receiving Title III 
fundinq for one of the four applicable products, actually participated In 
the Inquiry. These districts were enrolled In response to only a single 
general letter of Invitation, Even more Impressive Is that these 
participants completed and returned over 95* of the data collection 



2The earlier report (Hanson, Schutz, S '*^(:^77^s2rS;i^:^ 
test development and procedural «*P"ts ' n more deta l . See Hanson, 
ftlhr Mi-nuro and Ballev (1981) for a complete description of the test 
Behr, and other Instrumentation used In 

developiront procedure. Mctuai i.csi.3 omu « u,«««n MQ7R^ 

the study are described and reproduced In Bailey and Hanson (1978). 



13 



Instruments delivered to them. The actual number of pupils, classes, 
schools, and districts returning each study component Is given In 
Table 1. 

Insert Table 1 about here 



Note that the level of participation for one product, Dlstar 1, was 
limited to about 650 pupils from one state. By way of comparison, the 
participation levels for the other three products were substantially 
larger and representative with over 6,500 pupils from sU or seven states 
using each of them. 

Because of the small number of Dlstar 1 users and the rather large 
sample sizes required by the Inquiry design. It was decided at the outset 
of the Inquiry not to Include them as a product use group. Thus, the 
assessments distributed to them did not Include Items referenced to the 
three other products and their data were used only to fulfill the Title 
III evaluation requirements. However, the assessments distributed to the 
other three product groups did Include Items referenced to Dlstar 
outcomes. Thus, the data base that resulted Included Information from 
three groups of product users on Items and outcomes referenced to each of 
them as well as to a fourth set of Items referenced to Dlstar 1. 

PREVIOUSLY REPORTED FINDINGS ON THE EFFECTS OF EACH PRODUCT 
The Initial focus In analyzing the KRR study data was on fulfilling 
the Title til evaluation requirements within the project time 
constraints. Gl^en^'lhe extensive assessment data plus the supplementary 



11 



Information on pupil entry status and product Implementation, the 
required Title III results were easily generated. These results took the 
form of district level reports and state and national level aggregates of 
these results by each of the four products. The pr Imary 'concern In those 
reports was describing pupil proficiency (average percentage of Items 
correct) after Instruction on the outcomes referenced to the product In 
use In a district. 3 

Following the generation and distribution of these user reports, the 
first technical report was completed (Hanson, Schutz, * Bailey, 1977). 
Included within It were a number of analyses and findings which bear on 
the comparative evaluation of the products. These are summarized below: 

• Descriptions of study participants In terms of b'osoclal 
characteristics such as ethnic background, average ^-'"'W 
and status on national reading norms showed that each of the 
three product user groups Included schools representing each of 
the characteristics* categories. Further, there appeared to be no 
major Imbalances In the distributions on these characteristics 
across the three user groups. Put another way, using these 
characteristics alone for the three user groups, one would not be 
able to predict differential achievement or schooling advantage 
for any one group. 

• The rouah comparability of the three product S-'oups obse-ved 
on the blosoclal characteristics were ^"li^*;;,^^"^!^.';"*^^^^^^ 
Entry Survey tesc score distributions. The dl str buttons of 
class means on this measure were similar but "^^ J^*" 
the three groups. The highest grand mean was "^l^/^ ^ 
Alpha Time users followed by the Kindergarten users and then the 
Beqlnnlng users. 

• Product Implementation estimates obtained from teachers In terms 
of the number of lessons or units a class completed did show some 
differences across products. For the Alpha Time users, nearly 



3Detalled accounts of the actual report P;;^"^^^.^., 
Including an Illustrative district report are given In Hanson, Schutz, 

and Bailey (1977). 
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701 of the classes were In the highest quartile of product use as 
compared to 33% and 38!| for the Beginning and Kindergarten 
product userUy respectively. 

• The proficiency level of classes using each product on outcomes 
referenced to that product varied. However, high overall 
proficiency levels were obtained on the majority of the outcomes, 
although there were some notable differences. One yf these is in 
the percentage of product outcomes at the low and moderate levels 
of proficiency. For Alpha Time and Kindergarten user groups 
there were 3 and 1 outcomes, respectively, showing an average 
proficiency level of less than 601. This compares to 15 outcomes 
bejow that level for Beginning users. 

• An initial effort at establishing the Instructional sensitivity 
of outcomes was achieved by linking outcome proficiency to 
Implementation categories for each of the three product user 
groups. The average proficiency levels attained on each outcome 
by classes In each of four implementation guartiles was reported. 
The expectation was that classes Implementing more of a product 
should attain higher proficiency levels. These analyses showed 
this linkage could be found for some outcomes but not others. 
Overall, 27 of the 61 outcomes fW) showed patterns of 
increasing proficiency with more implementation. While these 
outcomes were not equally distributed across products, there were 
some outcomes that showed instructional sensitivity across each 
of the three product user groups. 

The initial KRR report ended at this point on a promising but as yet 

unconfirmed course toward a methodology for comparative product 

evaluation. It had served to nicely fulfill the Title Ml evaluation 

requirement In a useful way for both the districts and the government 

sponsors. Also, It did provide some evidence that the three groups of 

product users were roughly comparable In terms of baseline 

characteristics. Finally, It did Indicate that some linkage exists 

between the Implementation and proficiency measures, at least within, / 

products. As indicated previously, such linkage was considered as 

essential element in a methodology for a program-fair evaluation. 
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What remained to be accomplished was the development and application 
of a strategy for providing comparative Information on Instructional 
effects. As Indicated previously, such a strategy would Ideally provide 
Information for each product on both the Instructional sensitivity and 
proficiency level attained on Items and outcomes referenced to It and the 
other products. 

DESCRIBING AND COMPARING THE INSTRUCTIONAL EFFECTS OF COMPETING PRODUCTS 
To address the Issue of comparative product effects, the first step 
was to formulate a s^-.rategy for analyzing the extensive proficiency 
Information. The strategy centers around the three major Inquiry 
variables; entry proficiency. Instructional Implementation, and post- 
tnstructton proficiency. Data on these three variables wpre derived from 
each participating class using the Instrumentation described previously. 
These data were used as Input to a 2x3 matrix (see Figure 2). The two 
dimensions of this matrix correspond, respectively, to Instructional 
sensitivity (2 categories) and proficiency level (3 categories). It was 
used to classify each Item and outcome for each product-user group Into 
one of the 6 categories which result from crossing these dimensions. 



Insert Figure 2 about here 



To accomplish this classification, rules for determining inclusion 
In each category were derived. For the two Instructional sensitivity 
levels, a conwonallty analysis was used (see Figure 3). In this 
analysis, the Independent variables of entry survey mean score and amount 
of Instruction completed were regressed on the dependent variable, 
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average Item proficiency. The actual computations were carried out • 

stepwise, usinp both orders of Independent variables. The results 

provide a breakout of the portion of variance related to each of the 

Independent variables alone as well as shared. The rationale behind this 

analysis Is that for classes there should be a relationship between 

Instructional Implementation and proficiency after adjusting for any 

t 

differences prior to Instruction. The component of the commonality 
analysis ^*hlch represent this variable Is e In Figure 3, the port^lon of 
proficiency variance shared exclusively with the Instruction completed 
variable. 

\ 

\ ' 

Insert Figure 3 about here 

The actual results showed that this variance component (e In Figure 

« 

i 

3) accounted for between 0 and 51 percent of the variance across all 
Items. For reasons to be discussed later, a relatively low percentage 
value of 101 was chosen as a cutoff value for designating an Item as 
sensitive. This means If more than 10 percent of the variance of Item 
proficiency was shared exclusively with the Instruction con^leted 
variable, the Item was considered to be instruct ional ly sensitive. 

By choosing a low value, the strategy was to maximize the number of 
Items being designated as sensitive. The reason for doing so is tvnK>fold. 
First, the quality of the variable measuring product use as a general 
nwasure of product implementation was not knwn. Second, the Impact of 
curtailed Item proficiency variance Is known to be substantial. The 
restricted variance would be expected to depress the proficiency 
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Implementation correlation In the commonality analysis. Under these 

circumstances, a conservative criterion for Inclusion was considered most 

I 

appropriate. 

Forming the three proficiency categories In the decision matrix was 
straightforward. Item prof Iclency was determined by using the post- 
Instruction Item difficulties corrected for guessing, and forming a three 
category variable. The lowest category corresponds to chance plus 10% 
and the upper, 701 plus chance. All proficiencies falling between are 
referred to as In the "some proficiency" category. 

\ 

Illustrative Item Analyses 

The classification scheme was applied Independently to data on each 
study Item from the three product-user groups. The form of these 
analyses Is Illustrated In Table 2. This table presents the average Item 
proficiency and the commonality estimate associated with Implementation- 
proficiency variables for each of the three product user groups on four 
different Items, I.e., a total of 12 separate analyses. Using the 
proficiency level and commonality value, each Item Is assigned to a 
category (1 to 6) for each user group. The 12 analyses reported here as 
well as the others carried out In this Inquiry were based on data from at 
least 200 classes. 



Insert Table 2 about here 



Similar analyses were derived for the 278 Items on which each of the 
three product-user groups provided data. This resulted In a total of 83^ 
Individual analyses. I.e., three product-user groups by 278 Items. These 
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Item level data form the primary source of Information for this report. 
They are examined within and across several levels of aggregation, I.e., 
products, product-user groups, outcomes and Instructional areas, and 
Interpreted both In terms of their relevance to program-fair evaluation 
Issues and to the more general Issues of empirical curriculum Inquiry. 

Item Level Summaries 

Using the full set of Item classifications, overall summaries were 
derived for each product user group by product referenced. These data 
are given In Table 3. They show some Interesting differences both across 
products and user groups. Considering the products first, the results 
show A Items were most frequently placed In the high proficiency 
categories. Fully 89* of the classification were In either category 1 or 
I*, both reflecting high proficiency. This figure compares to 29* for B 
Items, 371 for K Items, and 271 for D Items. This suggests that the A 
Items were either considerably easier or were learned by the vast 
majority of pupils In the study. 



Insert Table 3 about here 

A related result Is the rather small proportion of the A Items which 
fall Into the sensitive half of the table. I.e., categories 1, 2, and 3. 
Only 16* of the A Item classifications were In these categories. This 
compares to 35*, 61*, and ^1* for the B, K, and D Item sets, 
respectively. Somewhat surprisingly, the D items were found to be 
sensitive more frequently than either the A or B Items. This would 
suggest that A Items were easier rather than learned. 
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Overall, the data show Items were classified In the sensitive 
categories, I.e., 1, 2, and 3, only abcut 1/3 of the time. This may 
reflect either an actual discrepancy between what Is Included In the 
product Instructional resources which were used to generate the Item 
pools and what Is taught, or It may reflect technical problems such as 
InsensltlvUy In the measure of Implementation used and/or high Item 
difficulty levels producing little Item variance. Both of these factors 
might Impact on the comnonal Ity analyses used to form this 
classification. 

To hedge against these possibilities, an effort was made to Include 
any Item In the sensitive categories that showed a relationship between 
product use and average proficiency level. This was the Justification, 
referred to previously, for using the value of lOt as the cutoff for the 
commonality value for Inclusion of an Item In the "taughf category. 

A summary of the item level data by user groups is given In Table h. 
These data show the results for the B and K user groups to be remarkably 
similar In terms of the percentages of all items assigned to each of the 
six classification categories. The largest difference between B and K 
users is In terms of sensitive Items and is reflected In the 7* 
difference in category 2, favoring the B users, i.e., 27t versus 201. 
This difference, however, is balanced out by the 51 difference in 
f itegory 1, favoring K users. I.e., 1?* versus 81. 



Insert Table k about here 



By the way of contrast, the B and K users differ substantially from 
the A users In terms of the percentage of Items classified within 
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categories. The most notable differences are In the large percentage of 
Items classified In the not sensitive categories for A users and also In 
the large number In the low proficiency categories. I.e., 3 and 6. For A 
users, 3i% of the Items fall Into these categories as compared to 19^ for 
6 users and 2\% of K users. In conjunction with the previous set of 
results (Table 3), these data indicate that A users performed at 
substantial ly lower levels of proficiency than B and K users on Items not 
reference^to their respective program. Also, there was little direct 
relationship between the levels of proficiency attained by A users and 
the amount of Alpha Time instruction received. 

While these findings emerge very clearly, the data have several 
limitations. One is that they Ignore differences between items In terms 
of the scope and Importance of the instruction they reference? i.e., all 
Items are weighted equally. An Item which measures a relatively complex 
skill for any kindergarten pupil, such as reading a sentence, Is treated 
as equal In Importance to an Item measuring an easier skill, such as 
knowledge of colors. 

A second related limitation Is the unequal numbers of Items used to 
represent products, outcomes, and instructional areas. The number of ^ 
Items vary from 58 to 85 across the four products. Such differences In 
the number of Items coupled with the fact that product users tend to 
score at higher levels on their own Items, further complicates the 
Interpretation of the Item level summaries* Under such conditions, 
products which Included larger numbers of outcomes and Items would be at 
an advantage In all cases In such comparisons. 
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Outcome level Sunwnarles 
' To obtain further Insight Into product effects, classifications of 
the Individual Items referenced to an outcome were aggregated to obtain 
an outcome level classification. In carrying out this aggregation, 
several technical Issues had to be considered. 

One Issue Is the classification consistency of Items falling under 
the same outcome. It would be Ideal If all the Item level analyses for 
an outcome and user group were assigned to the same classification 
category. This would make the outcome categorization match that of the 
Items perfectly. In practice, this happens Infrequently. The more 
typical result Is for most Items under an outcome to fall Into the same 
category with those remaining dispersed In categories around It. With 
such data, the modal Item category Is probably the best basis for 
categorizing an outcome and was used In this analysis. 

A second Issue, similar to that noted for Items, concerns the scope 
of Instruction referenced by an outcome. Some outcomes represent a 
substantial segment of product Instruction while others may simply be 
touched upon In Instruction. To some extent such differences In emphasis 
and scope are reflected In the number of Items Included for an outcome. 
However, It Is very difficult to take such differences Into account. 
Thus, m the outcome level analyses to follow, each outcome Is treated as 
"comparable." However, because the number of outcomes vary between 
products. I.e., from 10 to 25, «nd there Is the aforementioned expecta- 
tion that a user will fare better on outcomes referenced to the product 
in use rather than other products, there Is an Inherent bias in favor of 
the products with more outcomes, which should be kept In mind. 
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These factors considered, Table 5 presents the classification 
frequencies for outcomes by the product they reference for the three 
product user groups. There are different patterns for each set of 
prpduct outcomes. The A outcomes almost all fell In category I.e., 
not sensitive and high proficiency, for all three user groups. Only 11 
of 60 outcome classifications were not In category 4, and seven of these 
fell In category 1. Both 4 and 1 reflect high proficiency levels 
confirming that all three user groups attained high levels of proficiency 
on nearly all A outcomes. 

Insert Table 5 about here 

Results on the B outcomes show the most varlabllUy in assignment to 
categories across user groups. The largest frequency was again In 
category ^, with 28 of the 72 classifications falling In it. However, 
there were also significant numbers of outcomes classified in each of the 
five other categories. In fact, with one exception, each of the three 
product user groups had at least one B outcome classified in each of the 
six categories. 

One clearly distinct feature of B outcomes Is the substantial number 
classified in the middle and low proficiency categories (i.e., 2, 3, 5 
and 6). Over half of the outcomes were In these categories and were 
about equally distributed across each of the three user groups. B 
outcome classifications were quite similar for the three user groups In 
the not sensitive categories, i.e., ^, 5» and 6. However, consistent 
with earlier results, the A users show only 3 entries In the sensitive 
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categories. Thl. comperes to 11 outcomes In these c.tegorles for B users 

and 13 for K users. 

Focusing only on proficiency level, the results across the three 
user groups look very similar on the B outcomes. B, users as expected 
have a few more entries In the mid and upper proficiency categories. A 
users have a few less with the K users falling somewhere In between. 

The results on the K outcomes Is different but share som.. 
similarities with both the A and B results. Like the A outcci^es. f.w K 
outcomes were classified In the sensitive categories. However, like the 
B outcomes, the results .cross the high, medium, end low proficiency 
categories for all user groups are nearly Identical (eft.r eliminating 
the Instructional sensitivity distinction). Unlike B outcomes, which 
showed a substantial percentage of outcomes classified In the low 
categories (3 and 6) only 'l K outcome, out of a total of 33. was in a low 

proficiency category. 

The results for D outcome, differ In one clearly expected way from 
those of the other products, there were few outcomes classified in the 
high proficiency categories. Since the D outcomes were not referenced to 
the instruction provided to the A. B, and K user groups, this is to be 
expected, in terms of both level of proficiency and instructional 

sensitivity the K user, did slightly better than the B u which did 

slightly better than the A users, on the 0 outcomes. 

To sumwrize. the outcome level analysis showed substantial 
differences between the four products in terms of both sensitivity to 
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If^structlon and level of proficiency across. For A outcomes, low 
sensitivity and high proficiency was the dominant pattern. B outcomesi 
showed the most varied results with substantial numbers classified In 
each of the six categories. K outcomes were mostly cl-asslfled as not 
sensitive and In the medium and high proficiency categories. Finally, D 
outcomes fell mostly at the medium and low proficiency levels, and were 
about evenly divided between the sensitive and not sensitive categorUs. 

Instructional Area Analysis 

A summary analysis of the study data by instructional areas Is 
presented in Table 6. It gives the number of outcomes and Items from 
each product represented and their percentage of the total (in 
parenthesis below) for each Instructional area. Also, a detailed look at 
the proficiency levels attained by various subgroups of items and users 
within each instructional area is given. 



Insert Table 6 about here 



before proceeding to the proficiencies. It Is of Interest to examine the 
percentage of Items and outcomes falling under each instructional area In 
the four kindergarten reading products; At both the Item and outcome 
levels these data Indicate the first six Instructional areas of Word 
Meanings, Phonics, Word Reading, Sentence Reading, Oral Questions, and 
Letters subsume about 80% (76!^ and 831, respectively) of the con^oslte 
Instruction for the four products. Further, these six areas and one 
other. Writing, are the "common" areas, I.e., those represented by 
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outcomes and Items from two or more products. Given the emphasis on 
these six areas In the products, the focus for Interpreting the data will 
be on them. 

Within and across Instructional areas, the proficiencies In Table 6 
show differences between products. In Interpreting these differences. It 
Is helpful to have some kind of guideline* Because of^the way the data 
were gathered, 'driving a precise guideline for estimating statistical 
significance is difficult. Ideally, one would like to have some fixed 
value for a difference, e.g., 10* which would be the mimlnum difference 
to apply in I cases. Such a guideline Is possible In this case only by 
making some simplifying assumptions. 

Given that the minimum number of classes included in any user group 
estimate Is at least 250, and totally Ignoring the fact that each 
estimate Is based on multiple Items, confidence limits can be derived 
using standard formulas for percentages. These limits will be extremely 
conservative since nearly all proficiencies are based on multiple Items 
and Item/classes which usually number more than 250. Using the larger 
N's which would result from considering these two factors would certainly 
significantly reduce the size of the confidence limits. 

using 250 as the number of classes used to derive each percentage, a 
value of 1 5* Is a reasonable confidence band. That Is, a proficiency 
level exceeding another value by more or less than S% can conservatively 
be considered to be a statistically significant difference, it also 
might be argued that such a difference would be the minimum value of 
practical significance to school personnel. 
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Turnlnp to the data In Ti^Xa a clear distinction between results 
for the first six and last six instructional areas can be seen. Loolcinc) 
first at the last six instructional areas, only two outcome areas, 
Reading Questions (7) and Spelling (10) show any meaningful difference 
between user groups in overall proficiency. Reading Questions includes 
15 Items referenced to two D outcomes, and the average prof lei eiKles of 
all user groups were low and varied tittle i.e., from 261 to 321. 

The Spelling area showed similar low proficiency levels, it 
included three B items and the proficiencies ranged from 8^ to 22%, The 
large proficiency range was the result of the B users, >as expected, 
scoring higher than the A and K groups, i.e., nonuser>, on these Items. 

On the other four outcomes in this group. Affective Behavior (8^, 
Writing f9). Psychomotor Sl<li1s (11) and Mus.lc Awareness (12), the 
average proficiency for all three groups Is comparable--! .e. , within 
three percentage points. Also, user and nonuser proficiencies are nearly 
the same and overall proficiency Is quite high, i.e., from 70? to ?3I. 

Turning to the first six instructional areas, there Is a great deal 
more to be said since each instructional area presents a somewhat 
different situation. Lool<ing at Letters (6) first, the Interpretation is 
quite simple and straightforward. All three product user groups 
contributed outcomes and items, and extremely high levels of proficiency 
(about 83!^) were attained by all three product users on all item sets. 
Also user-nonuser differences are small for all Item sets. 

The results for the Oral Questions (5) area show another pattern. 
The overall proficiency level Is moderate (In the 67% to 701 range) with 
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little variability between either the three product user groups or the 

user-nonuscr groups. 

The Instructional areas of Word Reading (3) and Sentence Reading (M 
show similar and Important differences between all groups. For both 
these areas, Items and outcomes referenced to the B, K and D products are 
Included but not those of the A product. This absence of word and 
sentence reading Items points to a major difference between Alpha T.me 
and the other three products and Is surely a major reason for the lower 
average proficiency attained by A users In these Instructional areas. 

For word Reading (3), the proficiency differences across user groups 
within Item sets ranged from 131 to 65* and for Sentence Reading from 
31 to There were also large overall user group differences, ranging 

from 301 to ^.1* for Word Reading (3) and 111 to ^2% for Sentence Reading 
(M. There were also user-nonuser d ifferences. Both B and K users 
tended to perform significantly oetter .n their own Words and Sentences 
than on each other's and those referenced to DIstar. Overall, K users 
attained the highest proficiency levels In both Instructional areas with 
B users a close second on Word Reading (3) and a distant second on 
Sentence Reading (M. m both areas, A users we«J substantially lower 
than both of the other user groups. 

Phonics (2) Is an especially interesting area because it is the only 
Instructional area which Includes outcomes and Items from all four 
products. In spite of the fact that all products contributed Items, the 
proficiencies of the three user groups within item sets varied 
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substantially, I.e., from ^71 to 67^. Overall, B and K user 
proficiencies were BS% while the A users proficiency level was 
significantly lower at Sk%, User-nonuser differences ranged from 6% to 
111, showing substantial content/format differences in this area. 
Overall, the three user groups performed quite comparably on the large 
set of Dlstar outcomes and Items In Phonics (3). The average 
proficiencies differed by only 21, i.e., A (571), B (5B|), and K {$$%). 

The final and largest Instructional area by far In terms of both 
outcomes and Items Is Word Meanings (1). One-third of the outcomes and 
one-fourth of the Items are In this area. This Is In spite of the fact 
that one product, Dlstar, contributed no Items to Word Meanings (O. Of 
the three products represented, the B items were clearly substantially 
more difficult than those referenced to the A and K products. Average 
proficiency on the B Items was around 50%, while on the A and K items 
proficiency was greater than B0% in all cases. An examination of the B 
items Indicated that the discrepancy was probably due to the fact that 
some of the B Items combine less common concepts with more difficult Item 
formats. 

In spite of these distinctions, B users did not attain substantially 
higher proficiency on B Items. Generally speaking, there were relatively 
small differences between user groups on all" Item sets In this 
instructional area. Also, there were small differences between users and 
nonusers. I.e., 731 versus 711, showing little product-specific Impact. 

A final summary of the Instructional area data Is presented In Table 
7. In addition to the average proficiency attained by each product user 
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and the user-nonuser qroups, ft gives the average proficiencies attained 
by classes that conpleted all product activities. Thus, the Avg.-All 
columns estimate how prof Iclency would differ If the full set of product 
resources were utilized by all classes. Also Included In the table fin 
parentheses below the proficiencies) Is the percentage of Items found to 
be Instructional ly sensitive. This table probably represents the best 
overall summary of both what constitutes kindergarten reading readiness 
Instruction and the relative effects of the three products being 
compared. 



Insert Table 7 about here 



In terms of comparative product proficiency there Is a clear 
ordering of the three products? Kindergarten Program Is highest, followed 
fairly closely by Beginning to Read, Write and Listen, with Alpha Time a 
considerable way behind. Further, this ordering Is not changed much by 
the extent of Implementation of the products. Thus, the classes which 
completed all of Alpha Time actually Improved least In proficiency when 
compared to the average completion group. For B and K users, the results 
of Implementation were more clear cut. Both of the user groups showed 
substantial gains In most areas with greater Implementation. In several 
areas such as Word Reading (3), and Sentence Reading (5), «nd Writing 
(9), B users gained more with full implementation than K users. However, 
In other areas such as Phonics (2) and Reading Questions (7), K users 
gained more. 
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But how significant are the actual differences registered between 
products In terms of overall effects on pupil learning? Clearly, this 
question varies with the Instructional area being considered. If one 
uses the 5% average difference, K users produced a significant 
Improvement over B users In the areas of Sentence Reading ik) and over A 
users In the areas of Phonics (2), Word Reading (3) and Sentence Reading 
(M. 

By the same criteria, on the average, B users exceeded K users In 
the area of Spelling (10) and exceeded A users In the areas of Phonics 
(2), Word Reading (3), Sentence Reading {k) and Spelling (10). A users 
did not exceed the proficiency level attained by the other two user 
groups In any area. 

It Is of Interest to note how closely the empirical results 
correspond with the earlier analytical specifications for the products In 
terms of outcomes and Items. The two areas In which A users turned In 
the lowest relative proficiencies. I.e., Word Reading (3) and Sentence 
Reading (^), are areas In which the product was not represented by 
outcomes and Items. Likewise, the single highest relative proficiency of 
B users was attained In the Spelling (10^ area, one In which only It 
provided Instruction. 

This same kind of result did not hold, however, for A users In the 
Affective Behaviors, (8) and Psychomotor (11) areas. In spite of being 
the only source of Items In these areas, A users did not attain 
significantly higher proficiency levels than the other user groups. 
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The lnstruetlon.1 sensitivity pereent.ge. provide .nother dimension 
to the resultl, In T.ble 7. As noted In previous anelyses, these figures 
Show the A user qroup to h.ve many fewer sensitive Items. Further, the 
teble show, that this w.s espeel.lly true In the lnstruetlon.1 .re.s of 
Word Reedlns (3) .nd Sentence Heeding (M, -here there were not eny 
objectives referenced to AIphs.TIme. This Is exectly opposite to the 
results for B and K users, which had their largest percentage of 
Instructlonally sensitive Items In these areas. Overall, the latter two 
user group, had higher and basically con^Table percentages of sensitive 
Items. Variation In Item sensitivity across Instructional areas was 
quite substantial both within and across products. 

If these percentages are viewed ...Indicators of the extent of 
Instruction provided In a given area, they provide some Interesting 
Insights Into kindergarten schooling effects. If only those area, are 
selected for which somewhere near 50« of the Items were found to be 
sensitive, only three or four Instructlon.I areas emerge. These are Word 
Reading «.5«1. Sentence Reading (49*). and Reading Questions <53«). One 
other are., Phonics (37«), l« close. Other heavily Instructed areas such 
as vocabulary. Teacher Questions .nd Letters sh<« subst.ntl.l ly lower 
percentages of sensitive Items. 

interestingly enough, the .re.s In which highest percentages of 
sensitive Item, were found are also those In which substantial 
proficiency differences between product user groups were found. The 
convergence of these two kinds of Information provide, strong support for 
. simple but powerful proposition: products will only produce 



33 



ERIC 



30 

* 

differences tn areas in which the school is providing substantial 
instruction. Based on this Inquiry for the area of kindergarten reading 
readiness, these sicilis are primarily in the areas of Phonics, Word 
Reading, Sentence Reading and Reading Questions. 

In the other areas which were heavily emphasized by the products- 
such as Vocabulary, Teacher Questions and Letters"hlgher levels of 
proficiency were attained uniformly by all user groups but fewer Items 
were found to be Instructional ly sensitive. Perhaps such proficiency 
reflects general learning which occurs both In the Kindergarten classroom 
context and the overall environment of pupils. In any event there is 
little evidence that the products contribute differentially to learning 
in these areas. 

A corollary to this proposition concerns the differential effects of 
products. The study data show clearly that differences between products 
will only emerge In areas In which schools provide Instruction, and In 
which products differentially emphasize Instruction. This corollary 
assumes, of course, that product Implementation effort Is comparable. As 
product Implementation effort varies, effects vary directly and can serve 
to negate or even reverse expected differential effects of products. 

IMPLICATIONS 

The results of the Inquiry have Implications for a number of 
educational evaluation Issues. These Include the measurement of both 
educational product effects and implementation, two essential elements of 
any empirical program evaluation effort. Discussion of these elements 
provides background for the broader topic of evolving approaches to 
program evaluation and school Improvement. 
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Measuring Educational Product Effects 

The development of Instrumentation for measuring product, product 
comparative, and program characteristics was a central feature of the 
Inquiry. The findings support both prior and subsequent SWRL studies, 
I.e., Hanson and Schutz (1978), Hanson, Schutz, and Bailey (1981), 
Hanson, Bailey, and Molina (198l), Hanson, Behr, Meguro and Bailey 
(1980), and Hanson and McMorrls (1983), In defining new directions for 
the development pf product referenced measurement devices. 

The essence of this methodology Is the application of careful 
analytical methods to actual Instructional resources of products rather 
than to their objectives. This analytical work provides specifications 
for the generation of prototype Items which are then revised and 
validated through empirical Inquiries. The central element In the 
empirical revision process Is to link changes In measurement results to 
changes In Instructional resource Implementation. (Hanson, Behr, Meguro 

and Bailey, 1980). 

This process yields a form of assessment Instrument that differs 
both In form and substance from those usually employed In schools. The 
major difference In form Is the reduction In the length (number of Items) 
and scoring complexity (outcome scores). Separate scores need only be 
generated for broad but carefully defined Instructional areas rather than 
each potential objective or outcome. Traditional program objectives and 
outcomes are viewed as logical precursors for program and product 
development efforts. However, at the point at which actual product 
resources are available, these are discarded In favor of the resources as 
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the basis for the test specifications. After empirical verification, a 
much shorter test, with fewer scores to Interpret and composed entirely 
of Instructional 1y sensitive Items, Is the result. 

In single product as well as product*comparatlve Inquiries, such 
Instruments flreatly reduce the time and effort devoted to development, 
distribution, and collection of product effects Informatjon while 
Improving the Integrity of the Information provided. Assuming verified, 
product-specific Instruments were available for each of the four KRR 
products, for example, each class could then have tak^n one, two or all 
three of the competing product tests In addition to the one referenced to 
the product In use. In the same amount of time as was spent using an Item 
sampling approach. At the same time. Interpretation would be greatly 
simplified since variability due to sampling Is avoided. 

It should be emphasized that much of the economics In testing time 
and effort envisioned here rest on some Important findings concerning the 
various kinds of proficiencies students possess and those of Interest for 
program evaluation purposes. The KRR Inquiry sought to Identify only 
those areas of proficiency that were of significant scope to warrant 
reporting and that could be verified as linked to the product or products 
In use. The results Indicated that there are clearly both product- 
general and product-specific effects and that these effects can be best 
summarized In terms of six or seven Instructional areas, as discussed 
earlier In Tables 6 and 7. All of the rhetoric associated with 
Individual Items and outcomes Is not needed to summarize differences. 
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Given that the four products Involved and the sample of districts, 
schools, and classes In the Inquiry can be considered representative, the 
product-general effects caif really be characterized as kindergarten 
reading program effects, thus, program effects become defined 
operationally In terms of the major product- 1 Inked proficiencies 
displayed by students at the completion of a substantial schooling unit 

(semester or school year). 

Beyond kindergarten reading readiness program effects other 
contributors to student reading readiness proficiency such as other 
school programs and activities In the home and community exist. While 
these sources and the broader domains of proficiencies on which they 
Impact have not been addressed directly In this Inoulry, the methodology 
would appear to generalize to them In a straightforward manner. In the 
language arts area, for exasfip-le, the effects due to products In the 
program areas of reading, spelling, and writing could be examined 
simultaneously. 

If put In schools on an operational basis, an Important use of such 
effects Information would be to monitor factors affecting them over time. 
Such a school Information system, would Include Implementation measures 
In addition to the measures of the Instructional area effects and could 
serve as the basis for meaningful school Improvement efforts (Hanson, 
1978). This Issue will be discussed again later. The central point to 
be made here Is that through use of this testing methodology, measures 
can be devised which are very sensitive to school program effects, yet 
short and efficient to administer and Interpret. 
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f*rogram and Product tmplenwntat Ion Measurement 

Program Implementation measur&nent Is critical In understanding 
educational effects. In the KRR inquiry, Implementation measures played 
a central role since they Mere the primary basis for determining the 
sensitivity of Item, outcome, and Instructional area measures for each 
product. Although the practice of measuring program Implementation Is 
not well developed, this Inquiry as well as others, e.g., Hanson, Bailey, 
& Molina n98f), are providing iomt Insights, especially as to the 
Importance of product resource utilization as a central factor. Other 
related factors such as the amount of class time allocated, actually 
spent, and actually spent on Instructional tasks, can be considered to be 
logical precursors of product resource utilization. However, sorting out 
the exact relationships between the various Implementation factors and 
determining exactly how to utilize the Information they provide Is an 
Interesting and challenging task still to be completed. Th^ data 
sources « level at which Implementation factors are measured, and the 
constraints on the Instructional context all appear to affect the 
results. 

In the KRR Inquiry, the data source was teacher self-report, the 
level was the classroom, and the Instructional context was program and 
product-specific Other researchers have measured other Implementation 
factors, e.g., allocated and engaged Instructional time, at other levels, 
e.g., district, school, pupil, with other methods, e.g., direct 
observations, teacher logs, and In other Instructional contexts, e.g., 
program-specific, curriculum area specific. All provide different 
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perspectives on the In^lementatlon process and uses of Information about 

in KRR, Implementation Information was used to link up directly to 
measures of pupil achievement; the Implementation variable was considered 
a major Independent variable. In other Inquiries, Implementation 
variables have been evaluated alone and even treated as e dependent 
variable. A most promising future arrangement would be the treatment of 
product utilization as the major operational link to monitoring and 
Improving program effects In school Improvement efforts. The variable 
allocated product time could be used as the basis for altering 
utilization. 

To summarize, the role of Implementation measurement in research and 
improvement Is promising, but still emerging. KRR and related SWRL 
product inquiries Illustrate how It can serve as the major independent 
and operational variable of interest for both research on and improvement 
of the schooling process. As such. It provides the basis for both 
validating program effects and defining patterns of product use for a 
variety of audiences. 

An Evolving Approach to Program Evaluation 

The KRR inquiry, in spite of its limited scope, illustrates an 
emerging operational approach to educational evaluation. The focus of 
this approach Is slir^le: to Identify and explicate the effects, 
practices and costs of educational programs for the purpose of under- 
standing and Improving them. While this kind of rhetoric Is cocmwn In 
education, applications at a large enough level to demonstrate the 
veracity of proposed methods are rare. 

o 39 

ERIC 



3^ 

KRR showed clearly that methods for describing the direct effects of 
Instructional products and school programs exist. Further, these methods 
point to at least some of the Implementation practices behind the 
effects. 

Given measures of these two key elements and the operational context 
for deriving I nfprmat Ion on them, the requirements for obtaining 
meaningful cost Ih^formatlon Is also In place. Data on the Instructional, 
personnel, and other resources expended, for example, can be readily 
derived for a product (Hanson, I983). Other fixed cost components such 
as the facility are available and can be either prorated or Ignored 

i 

depending on the kinds of cost Information desired. With data on cos|s, 
Implementation factors and effects, very simple mode I $ linking the thrVe 
sets of factors to each other should be possible. The utility of these 
models for enhancing Improvement, program selection and program 
development efforts would, of course, depend on the strength 6/ 
relationships that emerge. 

Using the results from KRR provides an Illustration of the kind of 
models sought and Information they could provide. For the B and K 
products, there was substantial variability In Implementation which was 
linked to effects. I.e., the greater amount of product Implementation as 
measured b variables such as Instructional time spent and resources 
used, the greater the level of end-of-year proficiency. Given cost 

1 

figures could be derived for different Implementation levels, some useful 
options could be described, e.g., a simple cross tabular array which 
shows the costs associated with several levels of Implementation, and 
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several levels of effects. Such figures would provide the basis for a 
simple and meaningful description of programs In terms of the 
differential costs and effects of usinq alternative products. These data 
In conjunction with other Information sources about the products should 
put educational product selection, program development process and school 
Improvement efforts on a more Informed basis. 

As the enterprise of school, program, and product evaluation moves 
beyond textbooks and standardized testing, the possibilities for program 
planning and school Improvement will become reality. The combination of 
careful analytical and empirical analyses on the Instructional products 
used In various programs and factors Influencing their Implementation 
represents a promising means for achieving this advance. 



41 



38 



REFERENCES 

Bailey, J. 0., S Hanson, R. A. KRR Technical appendices * Los Alamltos, 
CA: SWRL Educational Research and Development, 1978* 

Coker, P. L., & Legum, S. E. Design of the Kindergarten Program Entry 
Survey (Technical Note 2-72-33). Los Alamltos, CAt SWRL ' 
Educational Research and Development, September 1972. \ 

Colcer, P. L., & Legum, S. E. An empirical test of semantic hypothase 
relevant to the language of young children (Technical Memorandgm 
2-7A-07) . Los Alamltos, CA: SWRL Educational Research and 
Development, Noven^er 197'^. 

Hanson, R. Bringing about basic changes In schools . Paper presented at 
annual meeting of American Educational Research Association, 
Toronto, Canada, 1978. 

Hanson, R. School program costs, effects and Improvement * Paper 
presented at annual meeting of American Educational Research 
Association, Montreal, Canada, 1983. 

Hanson, R. A., Schutz, R.E., & Bailey, J. D. Program-fair evaluation of 
instructional programs: Initial results of the kindergarten reading 
readiness inquiry, Technical Report No, 57 . Los Alamltos, CA: SWRL 
Educational Research and Development, 1977. 

Hanson, R. A., B Schutz, R. E. A new look at schooling effects from 
programmatic research and development. Chapter 7 In Making Change 
Happen ? in D. Mann, (Ed.) Columbia University Press, New York, NY, 
1978. 

Hanson, R. A., Behr, G., Meguro, B. T., S Bailey, J. D. Development and 
verification of instructional ly sensitive tests. Technical Report 
No. 69 , Los Alamltos, CA: SWRL Educational Research and 
Development, 1980. 

Hanson, R. A., Bailey, J. D., & Molina, H. The implications of criteria 
program placement decisions for the understanding and Improvement of 
schooling. Studies In Educational Evaluation ^ 2» 193-210, I98I. 

Hanson, R. A., Schutz, R. E., & Bailey, J. D. What makes achievement 
tests tick: investigation of alternative instrumentations for 
instructional program evaluation, unpublished report, Los Alamltos, 
CA: SWRL Educational Research and Development, 1981. 

Hanson, R. A., & McMorrls, R. Instructional sensitivity of alternative 
forms of achievement test items, unpublished paper, Los Alamltos, 
CA: SWRL Educational Research and Developement, I983. 



42 



39 



Maler, K., S SulUvan, H. Prooram-fal r assessment procedures Technical 
Memorandum 3-70-03. Los Alamltos, CA: SWRL Educational Research 
and Development, Hay 1970. 

Popham, W. J. Program-fair evaluation— summat I ve appraisal of 

Instructional sequences with dissimilar objectives. National 
Society for Programmed Instruction , 1969, 8, 6-9. 

Shoemaker, D. M. Evaluating the effectiveness of competing Instructional 
programs. Educational Researcher , 1972, 1(5), 5-8. 

Wiley, D. E., & Bock, R. D. Quasi -experimentation In educational 
settings: Comment. School Review , 1967, 353-367. 

Wolf, R. M. Program free testing. Research Memorandum. Los ^Ijj'^os, 
CA: SWRL Educational Research and Development, May 16, 1968. 



Instruct lon«1 Are« •nd Definitions 

T word H.«»l«fS--$«l«et* mustr.tlons/pletures corresponding to 
* th« (wanlns of spoken words end concepts. 

2. Phonlct-Selectf -ord pert corresponding to sitoken sounds. 

J. Word Medlng-Reeds .loud printed «rds preiinted elone or In 
sentences. 

Sentence «ewllnf--R«eds elood printed sentences. 

5. Orel duett lonS"Ans«ers questions el.pud ebout concepts, 
topics, storlet, etc. ^ 

6. Utter Identineellon-ldentlfles letters In e verlety of 
contents. 

f Bu««tlons--Reed» • short passe?* selects 

^' n?Il?r.uSll/rrds -hich ens-er questions .bout It. 

I. Affective •eh.vlor.-Olspleys effect In verlous specified 
classroom contexts. 

9, Wrltlng-^rlntt letters, wrds, and sentences. 

10. Spell lng--$eleets correct spelling of sl«iple «rds. 

II PsychoMtor Shi Ms-Olsplars physical coordination In 

specified dessroo* contexts. 
12. Iluslc Awareness-Selects appropriate musical Instruments 

based on their sou«ds. 



Alpha Tl 
2.3,5.10 

13 



me beginning Kindergarten 



6,7,9 
1,12 



8, II. IS. 16. 

17.18.19. 
20,21 

U 



2.H,8.II. 
12,13.1*. 
17.21,23 

7.9.15 
10.25 



26 

20.22,2*. 
29 



16 

19 
27 

28 



l,2,3.*,5 
6,7 



9 

11 
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f„>,r. .. ,..cr,p,l™- of l««rucU<»,.. .r... -d . M,, of -ro.uc, oo.co-., -Uh.„ .h.». 



Average Proficiency Level After Instruction 



Sensitivity to 
Product Impleroentttion 



Not Related 

(Coiwnonal Ity less 
than .10) 



Related 

(ConwnonaVlty more 
than .10) 



Low 

(Chance 101) 



VI 



III 



Hedlum 
(all between 
and hlQh) 



low 



11 



High 
(70% 4 Cha 



IV 



Fiaure 2. Six category classification scheme. Commonal I ty referred 
Figure 2. ^fx cat^g ^^^^^^ p,^f ufency variance accouned for by 

Instruction completed. (Corresponds to the quantity e In 

Figure 3.) 
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a dependent vir Uble-averige Item proficiency 

b independent varltble-ivere^e entry furvey fcore 

c Independent variable -amount of Instruction comp eted 

d portion of Item proficiency variance thared txclusWely 

with Independent variable b . , , 

c portion of Item proficiency variance shared exclusively 

with Independent variable c 
f portion of Item proficiency variance shared by variables 

b and c ^ * v 

d+c*f portion of Item proficiency variance accounted for by 

variables b and c 

FiQurc 3 Cownonallty analysis description carried out using class level 
F.gure 3. torwnona ^ty^ ^^^y ^^^^ ^^^^^ product-user groups. 
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Tt 



Number Returning 



le t 

•ch Study Component 



Alpha Time 
(A) 



Component 

Entry Survey 
Instruettonal 

Information Sheet 
School Information 

Sheet 
Assessment 



States Districts Schools Classes Pupils 

7603 



Beginning 
to Read, 
Write and 
Listen 
(B) 



Entry Survey 
Instructional 

Information Sheet 
School Information 

Sheet 
Assessment 



Kindergar- 
ten 
Program 

tK) 



Dlstar 
Program 
(D) 



Totals 



Entry Survey 
Instructional 

Information Sheet 
School Information 

Sheet 
Assessment 



111 
41 



18 
19 



Entry Survey 
Instructional 

Information Sheet 
School Information 

Sheet 
Assessment 



Entry Survey 
Instructional 

Jnformatlon Sheet 
School Information 7 ^03 

Sheet 

Assessment 7 



160 
137 



308 
2kk 





113 
117 




19 

19 




k02 
423 



899 2079'* 
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Table 2 

1 lustration of classification scheme 
applied to four actual Items. 



Item Description User Groups 



Product Alpha Time Beginning Kindergarten 



Example 


Items 


Reference 


f 


1 


c 


f 


1 


C 


p 


1 


c 


1 


IS 


(08) K 23 


29 


1 


i 




17 


2 


71 


23 




2 


FAT 


(11) S 27 


Ik 


10 


6 


33 


21 


1 


39 


i(1 


3 


3 


SECOND 


(05) A 21 


ik 


0 


5 


59 


2 


i 


76 


5 


k 


A 


Writing "1 
Have Fun" 


(28) D 05 


73 


0 


k 


68 


k 


k 


65 


0 





Legend 

Product Reference- item number Is given In parentheses* the letter 
refers to one of the four products (A - Alpha Time, B - Beginning, 
D - DIstar, K - Kindergarten) and the number Indicates the product 
outcome referenced. 

P - average Item difficulty across classes In the user group 

I - commonality estimate-percent of shared variance between 

class Implementation and proficiency variables (see Figure 3) 

C - Item classification category (1 to 6, see Figure 2) 
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Table 3 

Suirmary of Item CUssIf Icetfon Mth rowj^-centeges) 
by Product Referenced end User CroiiD 



Froduct 
Reference 



User 
Group 



Class If feet Ion Categories 




Proficiency Level 



Alpha Time (A) A 
(58 Items) » 

K 

Beginning (B) A 
(85 Items) B 

K 

I 

Kindergarten (K)A 

(60 Items) B 

K 



Distar (D) 
(75 Items) 



Totals 
(B3lt Items) 



A 

6 
K 



iensitive 
High Nedltta 
_J . 2 

0(0t) 0(0%) 
11(13*) M7i) 
9(161) M2%) 

3(U) . 7(Bt) 
2(21) 32(3BI) 
9(1U) ^'•(l^^) 

3(n) H«\ 

3(5t) tB(30l) 
11(180 2l(3St) 

3(U) 3(^1) 
6(81) 20(271) 
7(91) 19(25*) 



Low 
3 



High 



Not Sensitive 
Medium Low 
5 L- 



0(01) $0(861) 
0(01) «i2(72» 
1(21) *3(7^l) 

9(1 It) 25(29t) 
11(131) 17(201) 

2(3*) 15(25*) 
2(3*) 16 27* 
0(0*) 20(33*) 

1(1*) 15(29*) 
20(27*) IM19* 
13(17*) »6(21*) 



7(12*) 

1(2*) 

3(5*) 



1(2*) 
0(0*) 
1(2*) 



t3(15*) 25^^*) 

10(12*) 76%) 

\2{\k%) 22(26*) 

22(37*) 17(2BI) 
13(22JU B(13*) 
8(13*) OCO*) 

19(25*) 3i(W) 
7(9*) BOU 

9(12*) 11(151) 



67(8*) li.0(17*) B2(7*) 293(35*) 12M15*) 1^8(181) 
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Table k 

Number end Percent of AH Items 
elesslfJed In the %\x categories by user groups 



instruct lona i 
Sensitivity 


user 
jroups 


Proficiency Level 








High 


Hed 


Low 


Totals 


Sensitive 


A 

B 

K 


9(31) 
22(81) 

36(131) 


11(4%) 

74(27%) 

SS(20%) 


6(2%) 
31(12%) 
25(9%) 


26 
127 
116 




T 


67 


140 


62 




Not Sensitive 


A 
B 
K 


100(18%) 
97(171) ^ 
96(171) 


61(11%) 
31 (5%) 
32(6%) 


91(16%) 

23(4%) 

34(6%) 


252 
151 
162 




T 


293 


124 


t48 


565 * 


Totals 


A 
B 
K 


109(13%) 
119 14%) 
132(16%) 


72(9%) 

105(13%) 
87(10%) 


97(12%) 

54(6%) 

59(7%) 


278 
278 
278 




T 


360 


264 


210 


1'63M-^ 


*Percentages In 


each section 


based on the 


N In box [ 


5r 
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Table S 



i,t e.t.9orlMH<m. by product ind u.er «reup» . 
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Average prof fcfencles of the three product user groups (A, B e K) and 
user-nonuser groups (U, NU) on Item sets for Alpha lint (a), Beginning 
(b). Kindergarten (k), Olstar (d), and the totil (T). Also gives the 
number of Items (1) and outcomes (O) referenced to each product within 
the Instructional area. 




1. Word Meaning^ 



2. Phonics 





0 


1 


A 


B 


K 1 


U 


NU 


P 


0 


1 


A 


8 


K 


U 


NU 


a 


k 


21 


86 


86 


87 


86 


87 


a 


1 


6 


56 


67 


58 


56 


62 


b 


10 


2k 


^♦7 


52 


52 


52 




b 


3 


13 


k7 


57 


55 


57 ^ 


51 


k 


7 


21 


81 


82 


85 


85 


82 


k 


1 


6 


53 


61 


68 


68 


57 


d 
















d 


k 


33 


57 


58 


59 


«• mm 


58 


T 


21 66 
(32^)(2i»l) 


70 


72 


7k 


73 


71 


T 


9 58 
(1i»%)(2U) 


5^ 


59 


59 


59 


57 
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3. Word Reading 
P 0 I 1 A B 



a 

b 
k 
d 



2 12 

1 15 

2 18 



5 ^5 

{8!t)(l6%) 



13 
A2 

31 



30 
^♦9 
3^ 



23 
65 

3^ 



30 38 k] 



U NU 



30 
65 



18 
46 
33 



kS 33 



k. Sentence Reading 



a 

b 
k 
d 



1 9 
1 12 
1 6 



3 27 
(51) (101) 



6 



3 
6 

30 



21 
18 
3^ 



9 

kk 
k2 



11 22 32 



5. Oral ftgestlons 



6. letters 



53 



U NU 



21 



6 
12 
35 



^k 17 




P 0 1 


A B K 1 


U NU P 0 1 


A B K 


1 U NU 


a 3 5 

b 4 12 


70 68 69 
65 69 66 


70 69 a 2 6^ 
69 66 b 1 5 


Bit Bit 86 

82 85 81 

83 79 83 


Bk 85 
85 82 
83 81 


T 7 17 
(1U) (61) 


67 69 67 


69 66 T ^» 17 1 

(61) (61) 


83 83 8ii 

1 


Bk 83 



T»bl« 6 (continued) 



8. Aff>gttve Behavior 

! A B K 



U NU 




T 3 U 69 68 71 

(5^) W 



y 1 . Psychomotor Skills 

» A B K U m 



P 0 



b 
k 
d 



2 91 9^ 




SU 93 



2 91 9^ 9^ 



{2%) (It) 



a 

b 
k 



92 93 95 



93 93 



Sk 93 



t 2 92 93 95 

{2%) (U) 



93 93 
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Table 7 



SuTTmary of Proficiency and Instructional Sensitivity 
of Product Users on the Instructional Areas 











Product User Croups 


CofTipos I tes 


Instructional 
Area 


Number 
of Items 




A 

Avg.«Ail 


0 

Avg.-All 


Avg.-All 


Avg.-All 


1. 


Vocabulary 


66 


P 

s 


70-71 
(8%) 


72-76 
(25%) 


7^-78 

(271) 


70-78 

(21%) 


2. 


Phonics 


58 


P 
c 


5'*-57 
, (1^1) 


59-65 
(521) 


59-66 
(A5%) 


54-66 
(37%) 


3. 


Word Reading 


A5 


P 

c 


30-32 
(111) 


3B-'i9 
(671) 


i»l-50 
(581) 


30-50 
(45%) 


k. 


Sentence Reading 


27 


< 


11-12 

W) 


22-35 
(7k%) 


32-42 
(701) 


11-42 
(49%) 


5. 


Teacher Questions 


17 


P 

c 


67-68 
(181) 


69-75 
i2k%) 


67-73 

(W) 


67-75 
(29%) 


6. 


Letters 


17 


P 

C 

w 


83-85 
(181) 


83-86 
(2k%) 


8A-88 
(291) 


83-8B 
(24%) 


7. 


Reading Questions 


15 


P 
S 


26-28 
(01) 


29-^0 
\o7l) 


32-'»5 


26-45 

(53%) 


S. 


Affective Behavior 


U 


P 

s 


7^-7^ 
(01) 


73-71 
(2 It) 


73-74 
(211) 


7i-7*« 
(14%) 


9. 


Wr i t ing 


12 


p 
s 


69-73 
(81) 


68-7^ 
(25%) 


71-73 
(01) 


68-73 

(n%) 


10. 


Spel 1 f ng 


3 


p 
s 


8-11 

m 


22-32 
(33t) 


11-8 
(01) 


8-22 

(n%) 


n. 


Psychomotor 


2 


p 
s 


91-9^ 
(01) 


5U-Sk 
. (01) 


94-97 
(01) 


91-97 
(0%) 


12. 


Husic Awareness 


2 


p 

s 


92-90 
(01) 


93-9^ 
(01) 


95-97 
(01) 


90-97 
(0%) 



p - averigt proficiency 



S percentage of Items sensitive 
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APPENDIX A - Product Outcome Descriptions 
A - Alpha Time 

B - Beginning to Read, Write and Listen 
K - Kindergarten Program 
D - DIstar I 
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A - ALPHA TIME (New Olmentlons) 



Outcome Instruct ional Common 
Number Area Items- 



Name 



Description 



1. 



2. 



3. 



8 



5. 



6. 



7. 



5 
5 



1 
3 



8. 



8 



9. 



Overall Letter Skills Identify letters in 

Isolation and word 



Word Analysis/ 
Vocabulary 



Word Analysis/ 
language 



Simple Recall Skills 



Sequence 



Ha in Idea 



Prediction and 
Inference 



Reading and 
Literature 



Interpretive Skills 



context s; 

Identify colors, shapes, 
positions, sizes, and 
•mounts. 

Use appropriate vocabulary 
to answer questions about 
a picture or story, select 
a letter that is the first 
letter of the name of a 
pictured object, and name 
objects in a picture. 

Recall simple facts 
deal ing with who, when, 
where, why, wh*it, and how 
of a teacher read story. 

Identify the correct 
sequence of ordered events 
following a teacher read 
story. 

Identify the best title 
of a teacher read story. 

Explain a logical Inference 
drawn from a picture, 
provide logical consequences 
of an Incomplete story, 
and supply the rhyming word 
that completes a poem. 

Demonstrate personal 
pleasure, appreciation, 
and interest in Ute.'ature; 
demonstrate care for books; 
and use books for specific 
purposes. 

Interpret the moods or 
feelings of characters 
In story situations. 



er|c 
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Outcome Instructional 
Number Area 



10. 

n. 



1 

8 



rOmmon 
Items 

1 

2 



Name 

Following Directions 
Dramatic Play 



Description 

Follow almple directions. 

Give dramatic expressions 
to pantomime, and dramatize 
lines or words so that the 
audience can identify the 
mood, amotion, or story 
being conveyed. 



1 7 
1 ^ * 


6 


3 


Visual Hemory 


Identify the letter of the 
AinK*h#t when oresented with 
the letter name or a word 
beginning, with the specified 

1 *t fur. 


* ^ • 


2 


6 


Auditory Hemory 


Identify letter sounds in 

It tmni # words • 


m 


9 


i 


Psychomotor Skills 


Write the letters of the 
alphabet In upper and lower 
when oresented with en 
example of the letter or 
the name of the letter. 


15. 


8 


2 


The Need to lelong 


Participate In group 
activities and interact 
comfortably with new people. 


16. 


8 


2 


The Need to Achieve 


Achieve personal goais 

and complete assigned tasks 

Independently. 


17. 


8 




The Need for Love 
and Affection 


Accept affection from 
others. 


18. 


8 




The Need to be Free 
from Gui It 


RecognUe that mistakes 
are to be expected. 


19. 


8 




The Need for Self 
Respect 


Express opinions freely 
to the teacher or 
classmates. 


20. 


8 




The Need to be 
Free from Fear 


Share feelings with 
classmates. 


21. 

ft 


8 




The Need for 
Understanding 


Listen attentively. 
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B - BEGINNiHG TO READ, WRITE AND LISTEN (Llpplncott) 



Out com Instruct ional Common" 
Number Areas Items 



Ham 



Description 



I. 

2. 

3. 

k, 

5. 
6. 

7. 

8. 

9. 

10. 

n . 

12. 

13. 

]k. 

15. 



I 
2 
3 



1 
I 
2 



3 
6 

3 

6 
3 

! 

3 
1 



Colors 

Shapes 
Amounts 
Posit Ion 

Visual Discrimination 

Al phabet 

Letter/Sounds 

Synonyms 
Blending 



Word Recognition 
(Memory Words) 



Vo-abulary 



Inference 



Parts of Speech 



Rhyml ng 



Vowel /Consonants 



ERIC 



Identifying basic colors 
and those produced by 
their mixture. 

Identifying the shape 
of objects. 

identifying objects that 
differ in amount. 

Identifying the position 
of objects. 

Distinguishing between 
fami 1 lar objects including 
letters and words. 

Identifying upper and 
lower case letters of the 
alphabet . 

Associating approriate 
sounds with the letter 
tyn^ol . 

Identifying synonyms for 
words. 

Putting together letter 
sounds to form words. 

Recognizing words introduced 
in the program that are 
not blended. 

Understanding the meaning 
of coimion words. 

Completing a phrase or 
sentence with a logical 
word. 

Identifying verbs and 
adjectives. 

Identifying and forming 
rhyming words. 

Distinguishing vowels 
and consonants. 



59 



f 



Outcome Instructional 
Number Areas 



Common 
Items 



Description 




17. 
18. 

19. 
20. 
21. 



22. 
23. 

2k. 

25. 

26. 

27. 
28. 

29. 



10 

5 
1 



5 

1 



k 
U 
12 



3 
2 
1 



3 
3 



9 
2 
2 



Punctuation 
Syllables 
Spel 1 Ing 

Picture Interpre- 
tation 

Classification 



Story Interpretation 
Sequence 

Comprehension 
Word Attack 

Sentence Reading 
Manual Dexterity 
Husic Awareness 



Oral Language 
Experience 



Forming the upper and 
lower case letters, either 
singularly or in words. 

Distinguishing punctuation 
marks. 

Dividing a word into 
lylUbles. 

Understanding the meaning 
of common words. 

Interpreting the meaning 
of pictures. 

Classifying common objects 
according to specific 
dimensions such as 
iweet-sour . 

Interpreting the meaning 
of a story. 

Properly ordering a 
specific set of objects, 
experiences, etc. 

Answering specific 
questions about a 
teacher read story. 

Reading words composed 
of already learned word 
elements or letter sounds. 

Reading sentences of 
already learned words. 

Demonstrating manual 
dexterity in simple tasks. 

Reacting to musical 
Instruments and to the 
music made by them. 

Telliiig a short story 
•bout an experience. 
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K- KINDERGARTEN PROGRAM (Ginn and Company) 



Outcome 
Number 

1. 



2. 

3. 

k, 

5. 
6. 



Instructional 
Areas 



Conmon 
items 



Name 



7. 



8. 
9. 

10. 



3 
2 



Colors 



Sizes 

Amounts 

Shapes 

Positions 
Pre-Math 



Pie-Readtng 



15 Words 

6 Word Elements 

12 Sentences 

6 Letter Names 



Description 

Select and name examples 
of the concept, color 

Select and name examples 
of the concept, size. 

Select and nan« examples 
of the concept, amount. 

Select and name examples 
of the concept, shape. 

Select and name exbmples 
of the concept, position. 

Select and name examples 
In the area of pre- 
mathematics (e.g., equal, 
different) . 

Select and name examples 
In the area of pre-reading 
(e.g. . f I rst [in time] , 
after [In time], letter, 
sound) . 

Identify program words. 

Identify the sound of 
program elements. 

Read aloud each word in 
sentences composed of 
program words. 

Identify the letters 
of the alphabet. 
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D - OlSTAR I READING (Science Research Afsocletes) 



)utcome Instructional 
WumbeiL Areas 



»onvnon 
Items 



Nanie 

12 Symbol— Action Games 



2. 



3. 



5. 



6. 



7. 



12 



Blendlng"Say 
It Fast 



Rhyming 



9 Blending— Spelling 
by Sounds 



3 Writing 



Sound Recognition 



Word Recognition 



8. 



6 



Word-Attack Skills 



9. 



ERIC 



Reading Stories 
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Description 

Identify and repeat 
pictured or teacher 
presented sequences of 
bodily movements. 

Say a word at a normal 
speaking rate when it 
Is presented slowly with 
pauses between the parts. 

Hake up words that rhyme 
with a given word by 
substituting new Initial 
sounds. 

Spell a word by sounds 
(say one sound at a time) , 
without pausing between 
the sounds, when the teacher 
says a word at a normal 
speaking rate. 

Write a symbol, word, 
phrase, or sentence by 
tracing the dotted 1 ines 
when given a printed sample 
or write one freehand that 
was previously traced. 

Say the sound represented 
by a symbol when given a 
printed symbol . 

Read aloud either a word 
composed of previously 
taught sounds or a common 
word that does not have a 
regular spelling. 

Read words that have double 
confonants; initial or final 
consonant combinations, or 
comnon endings; and read a 
series of words ending in 
rhyming patterns. 

Read stories of increasing 
length and complexity both 
with and without typographical 
aides. 



Outcome 
Number 



Instructional wommon 
Areas Items 



10. 



Hame 

i Answering Questions 



n. 



Comprehension 



Description 

Hake an independent 
written response when 
given an oral direction 
based on stcilis previously 
taught. 

Answer questions from 

the teacher that demonstrate 

reading understanding. 
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APPENDIX B - Statistics on Common Items by 
Instructional Area, Product 
Referenced and Product-User 
Group 



Explanation of Entries In Appendix B Using 
Two Illustrative Items 



item Description 




User Groups 




Product 

Example Items Reference 


Alpha Time 
P 1 C 


Beginning 
P 1 C 


Kindergarten 
P 1 c 



1 

2 



IS 
FAT 



(08) K 23 
(n) B 27 



29 1 i 

2k 10 6 



kS 17 2 71 
33 21 3 39 



Legend 



Product Reference- 1 tern number Is given In parentheses, the etter 
refers to one of the four products (A - Alpha Time, B - Beginning, 

0 - Dlstar, K - Kindergarten) and the number Indicates the product 
outcome referenced. 

P - average Item difficulty across classes In the user group 

1 - conwonallty estimate-percent of shared variance between 

class Implementation and proficiency variables (see Figure 3; 

C - Item classification category (1 to 6, see figure 2) 
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